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[57] ABSTRACT 

The disclosure is directed to an improved method and 
apparatus for generating a sequence of video frames 
representative of three-dimensional animation. A plu- 
rality of key frames are stored, each key frame including 
a common figure having one or more joints, and each 
joint having associated therewith a set of vectors defin- 
ing a limb. Each joint is defined in each frame by opera- 
tor-controllable parameters which determine the three- 
dimensional position, rotational orientation, and scale 
factors of a local coordinate system in which the limb 
vectors are placed. A plurality of in-between frames are 
generated, the in-between frames including the com- 
mon figure having one or more joints and limbs corre- 
sponding to the joints and limbs of the common figure 
in the key frames. The parameters of the joints of the 
in-between frames are obtained by interpolating in three 
dimensions, the position, rotational orientation, and 
scale factors of the corresponding joints of the key 
frames. In the preferred embodiment, the joints of each 
figure are arranged in hierarchical order, and the posi- 
tional coordinates and rotational orientations of the 
local coordinate system for a particular joint are deter- 
mined with respect to the local coordinate system of the 
next higher joint in the hierarchy. Also, the operator 
can control the interpolation during display of the in-be- 
tween frames, so as to change the motion of a figure 
limb. 

22 Claims, 11 Drawing Figures 
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time and effort that is necessary to generate a series of 
THREE DIMENSIONAL ANIMATION animated frames. Particularly when dealing with three- 

dimensional animation, however, the automatic genera- 
BACKGROUND OF THE INVENTION tion of in-between frames can tend to result in unnatural 

This invention relates to the field of machine genera- 5 motion of the three dimensional figures, 
tion of animation and, more particularly, to a method It is among the objects of the present invention to 
and apparatus for generating sequences of frames on a provide a method and apparatus for generating frames 
two-dimensional screen that are representative of three- of three-dimensional animated sequences which have 
dimensional animated figures. natural-looking motion. It is a further object to provide 

Animated films are in widespread use for educational 10 an animator with a high degree of flexibility in obtaining 
and entertainment purposes. Animated films of reason- the desired appearance of motion of figures with mini- 
ably high quality have traditionally been very expensive mal labor, 
to make. The initial creative work includes generation 

of a basic story, a sound track, and basic character de- SUMMARY OF THE INVENTION 

signs. Typically, a schedule is generated which de- « The present invention is directed to an improved 
scribes the movements and timings of each scene to be method ^ apparatus for generating a sequence of 
produced, and layout sketches and "key" drawings are ^ presentational of three^imeSonal ani- 

madewhich define the overaU design of each scene. It mation . ^ a( Jj danoe ^ ^ method of ^ ^ 

has been generally observed that at this point most of „ . ... ri . ^ , ,7 J 

the creative design work has already been done. The 20 tion > a plurality of key frames are stored, each key frame 
so-called "key" drawings or frames are still frames ^Judmg a common figure having one or more joints, 
which show extremes of action. In the traditional ani- and ^ h _ JOmt ^vmg associated therewith a set of vec- 
mation procedure, people known as "m-betweeners" J 0 " defhm *g a limb.Each joint is defined m each frame 
are employed to sketch frames which fill in the action by operator-controllable parameters which determine 
between key frames. After in-betweens are sketched by 25 ti» three-dimensional position, rotational orientation, 
hand, the hand-drawn pictures are typically transferred 311(1 ? cale fact ors of* a local coordinate system in which 
onto celluloid-acetate sheets and are then painted in by the ! imb vectors are placed. A plurality of in-between 
hand. Much labor is involved in obtaining the in-be- frames are generated, the in-between frames including 
tweens, which typically comprise the majority of toe common figure having one or more joints and limbs 
frames of the final work product These tasks render the 30 corresponding to the joints and limbs of the common 
production of animated films an extremely expensive figure in the key frames. The parameters of the joints of 
proposition. An advantage of the traditional animation the in-between frames are obtained by interpolating, in 
process, however, is that the animator has virtually three dimensions, the position, rotational orientation, 
complete artistic freedom and control over the resulting and scale factors of the corresponding joints of the key 
film; i.e. anything that is drawn can be made to move in 35 frames, 

a desired fashion so natural-looking motion can gener- In the preferred embodiment of the invention, the 
ally be achieved. joints of each figure are arranged in hierarchical order, 

In recent years, the field of machine generation of and the positional coordinates and rotational orienta- 
animation has made great strides. Advanced computer tions of the local coordinate system for a particular joint 
graphics techniques have improved the speed and qua!- 40 m determined with respect to the local coordinate 
ity of computer-generated animation. Computers can system of ^ next higher joint in the hierarchy. Also, in 
aid artistic animators m many ways, including auto- accordance with the preferred embodiment, the opera- 
mauc generation of in-between frames by interpolation tor ^ caatx6L ^ interpolation during display of the 

of t3^LwS^ ,VC key A< in-between frames, so Tto change the motion of a 

of this type is described, for example, in an article eno- 45 r 12ure k—l 

tied "Towards a Computer Animating Production -pl^w r^*,, , , , f4 , . ... 

Tool" by N. Burtnyk and M. Wein which appeared in J^* 6 ' fcatUfeS and ^ 
the proceedings of Eurocomp Conference; Brunei - ^ecoine inore apparent from the following detailed 
United Kingdom, May 1974. description when taken m conjunction with the accom- 

It is known in the computer graphics art that three-di- 50 panying ^^S 8 - 
mensional representation of animated figures can be BRIEF DESCRIPTION OF THE DRAWINGS 
stored as a set of three dimensional points, and appropri- 

ate transformations can be used to automatically com- na 1 15 a block of an apparatus m accor- 

pute the . projection of the three-dimensional figures dance WIth 811 embodiment of the invention, and which 
onto a two dimensional surface, so that two dimensional 55 ^ US ^ A to P™ 0 ^ method of the invention, 
computer generated animation images can be presented FIG * 2isa fl ow diagram which describes the general- 
on a conventional two-dimensional screen. Three di- ^ procedure for producing three-dimensional anima- 
mensional figures have also been represented in joint uon on a two-dimensional screen, including the tech- 
and limb configuration. mque of the invention. 

Representative prior art patents relating to tech- 60 3 is a flow diagram of the routine for forming 

niques such as computer-generation of in-between skeleton files for a sequence of animation to be gener- 
frames and three-dimensional animation are as follows: ated. 

U.S. Pat Nos. 3,364,382; 3,523,389; 3,585,628; FIG. 4, which includes FIGS. 4A and 4B, illustrates 
3,603,964; 3,700,792; 3,723,803; 3,747,087; 3,792,243; a joint and limb in a local coordinate system, and shows 
3,883,861; 3,917,955; 3,885,096; 4,017,680; 4,127,849; 65 the manner in which a joint and limb are represented 
4,189,743; 4,189,744; 4,200,867; 4,213,189. and can be manipulated. 

As above-stated, the automatic machine generation of FIG. 5 is a flow diagram which illustrates the routine 
in-between frames can greatly reduce the amount of for displaying frames of figures represented by joints 
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line drawing system 70, and includes processor 71 and 
line drawing display 72. A suitable vector processing 
and display system is the Evans and Sutherland Com- 
pany "Multi Picture System". A raster-based system 

5 which performs similar functions is described in an 
article entitled "Coprocessing to Ease The Graphics 
Burden" which appeared in the July, 1982 issue of 
"Computer Design". The system 70 is coupled via the 
bus 30A to the other equipment A video tape recorder 

10 55 and a television camera 65, may also be provided in 
the system. 

Referring to FIG. 2, there is shown a generalized 
flow diagram of a typical operational procedure for 
generating a sequence of frames of three dimensional 
animated scenes for display on a two-dimensional 
screen. It will be understood, however, that variations 
in this typical procedure can be employed, consistent 
with the principles of the invention, depending, inter 
alia, upon the type of animation to be produced, the 



15 



and limbs in three dimensions on a two-dimensional 
display. 

FIG. 6 is a flow diagram of the routine for forming a 
frame, such as the key frame. 

FIG. 7 is a flow diagram of the routine for generating 
in-between frames. 

FIG. 8 is a flow diagram of a routine for modifying 
the motion of a limb during display. 

FIG. 9 shows an interpolation curve and a modified 
interpolation curve. 

FIG. 10 is a flow diagram of the routine for modify- 
ing the interpolation curves. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENT 

Referring to FIG. 1, there is shown a block diagram 
of an apparatus 10 in accordance with an embodiment 
of the invention and which can be used to practice the 
method of the invention. An operator 15, who is typi- 
cally, but not necessarily, an animator of some artistic 20 source of component figures, and operator preference, 
ability, works at a console which includes the devices The block 211 represents the inputting and storage of 
illustrated within the dashed enclosure 20. In the pres- limb files, which, if already existing, can be keyed in via 
ent embodiment, the console includes three types of the keyboard 22, or can be drawn in via the data tablet 
input devices whereby the animator can input informa- 21, or by using the joysticks and cursor control. In the 
tion to the apparatus 10, viz. a data tablet 21, a keyboard 25 present embodiment, each limb is represented, in known 
22, joysticks 23 and monitors on which information is fashion, as a set of vectors in three dimensions which 
displayed, viz. monochrome display 23 and optional define the limb shapes to be used on the figures in the 
color display 25. sequence of animation. The form of representing the 

The devices in the console 20 are coupled via a bus vectors and limbs is treated hereinbelow. It should be 
30A to control circuitry 30. In the present embodiment 30 noted that this part of the procedure is not strictly nec- 
the novel functions of control circuitry 30 are imple- essary to perform at this time (since limbs can be pro- 
mented by an appropriately programmed general pur- duced after the skeleton of joints is formulated, as de- 
pose digital computer, for example the model VAX scribed hereinbelow), but in many practical situations 
1 1/780 or the model PDP-11/45 manufactured by Digi- one can utilize previously stored three dimensional 
tal Equipment Corp. of Maynard, Mass. However, it 35 vector representations of hmbs available from a "limb 
will be understood that alternate means, such as a spe- library". Also, it is usually convenient to generate limbs 
cial purpose computer or other suitable circuitry having beforehand for use during the sequence of animation to 
logic and storage capabilities could be utilized to be generated, it being understood that certain limbs can 
achieve the desired functions. In conjunction with the be utilized a number of times in the various figures of 
general purpose digital computer 30 there is provided 40 the animation. The representative limb to the left of 
memory which includes fast access bulk storage such as block 211 is a simple robot foot 
disk memory 52, and random access storage labelled The block 212 represents the formation of a skeleton 
RAM 53. Typically, at least a portion of the random file for each figure in the sequence of animation to be " 
access memory will be included within the general generated. As previously indicated, each figure includes 
purpose computer 30, and it will be understood that the 45 one or more joints, and each joint has associated there- 
amount and type of each storage medium or alternative with a limb. Each joint is defined in each frame by 
storage can be selected by one skilled in the art. operator-controllable parameters which determine the 

An optional frame storage means 60 is provided and three-dimensional position, rotational orientation, and 
is coupled, inter alia, to the control circuitry 30 and to scale factors of a local coordinate system in which the 
the color monitor 25. The frame storage means is of the 50 limb vectors are placed. The joints of each figure are 
type known as a "frame buffer" manufactured and sold arranged in a hierarchical order, and the positional 
by Genisco Computer Company of California. Essen- coordinates and rotational orientations of the local co- 
tially, the frame-buffer 60 is an addressable digital mem- ordinate system for a particular joint are determined 
ory which stores a frame of video information. The with respect to the local coordinate system of the next 
frame buffer 60 is coupled to the control circuitry 30 via 55 higher joint in the hierarchy. The information defining 

these parameters for each joint will be described in 
further detail hereinbelow. A simplified skeleton figure 
is illustrated to the left of block 212 of FIG. 2. In this 
sketch, the joints are represented by small circles and, 
60 for ease of illustration, straight lines are shown as con- 
necting each joint to its "parent" joint; Le., the joint of 
next highest priority in the joint hierarchy. In the 
sketch, the joint Jl is the highest priority joint for the 
figure, so the motion, rotation, or scaling of the joint 



the bus 30A so that the control circuitry 30A can typi- 
cally be utilized to address any desired pixel in the 
frame buffer, for example to interrogate or read the 
pixel value contained therein or to write in a new pixel 
value at any point 

In the present embodiment of the invention, it is con- 
venient although not required, to employ a commer- 
cially available processor for performing certain matrix 
transformations and display tasks which could alterna- 
tively be performed by general purpose processing and 65 parameters for Jl will serve to move, rotate, and scale 



display circuitry. In particular, a three-dimensional line 
drawing display system with coordinate transformation 
and clipping hardware is collectively referred to as the 



the entire figure in the world coordinate system. The 
joints J2, connected to joint Jl as their parent, are the 
* next lower order joints in the joint hierarchy. There are 
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three such child joints J2, designated as J2A, J2B, and 
J2C. Also, the joint J2A is a parent of three more child 
joints designated as J3A, J3B, and J3C. 

The routine for forming and storing the figure skele- 
tons is shown in FIG. 3. The block 311 represents the 5 
identification of the skeleton file for the particular fig- 

ure whose skeleton is to be generated. A joint is identi- V****. ^ m ' and respectively represent the x, 

fied to be input (block 312) and its connection in the £ ° r * move " ^° rdinates of ^ ori & n of 

hierarchy is indicated, eg., by identifying its parent in ^J^f ^ ordma ] e the point at which the 

joint If desired, thelcelefon can be d2yed duTg 10 ^^1° " f H ^ ^ T FT* 

this operation (see display routine of HG 7) Initial x, y S^i^S ^ f Sy *T 

. „ \ . . A ' , ' 7 pnonty joint) or to the local coordinate system of the 

and z co^amates of the jom posidon are stored, as ncxthigher p ^ ority ^ hierarchy of joints. The 

represented by the block 313. Initial rotational onenta- values Txf Xy> and respective rotational an- 

tion values and scaling factor values can then also be 15 ^ muad the x, y, and z coordinate axes of the local 

stored (for example, zero degrees rotation and unity coordinate system (again, with respect to the orienta- 

scalmg factor, as initial values) as represented by the tions of the coordinate axes of the next higher priority 

block 314. The meaning and purpose of these values are joint coordinate system). The values $ x , s» and s, are 

described further hereinbelow. The next joint of the scaling factors for the x, y and z directions of the local 

figure can then be added (re-entry to block 312). Addi- coordinate system. A value of unity is used to represent 

tional figure skeletons can be formed and stored by the normal scale factor in each dimension, and higher or 

repeating the procedure. lower numbers can be used to stretch or compress the 

Referring again to FIG. 2, the block 213 represents limb associated with the local coordinate system in any 

the portion of the generalized procedure wherein indi- 25 direction. 

vidua! figures of a frame are formed, so that the desired FIGS. 4A and 4B are useful in understanding the 

poses are obtained for the key frames of the animation manner in which the joints and limbs are represented 

sequence to be generated. This routine is described in manipulated. In FIG. 4A, there is shown the coor- 

conjunction with FIG. 6. The figure sketched to the left dinate axes for an exemplary joint, and a limb, 

of block 213 is useful for initial illustration of how the 30 ( s ^ own in dashed line) of vectors associated with the 

limbs, designated with notation similar to the joints with J omt The matrix values for the joint J/ is as follows: 
which they are respectively associated, except that an L 
is used instead of a J, are stored in conjunction with the 
skeleton file. Thus, each joint of the skeleton file for a 
figure has stored in conjunction therewith a limb file (or 
identification of the limb file) for the particular limb that 

is selected by the operator for use at a particular joint As seen in FIG. 4A, the origin of the local coordinate 

position of the figure. system for J,* is at the point (10, 10, 10) in the main or 

The key frames are stored (block 214) and, under 40 "world" coordinate system (which, in this example, is 

operator control, in-between frames can be automati- the coordinate system of next higher priority in the 

cally generated (block 215) in accordance with the hierarchy). Also, the x, y, and z axes are aligned with 

principles of the invention, and as described in conjunc- the axes of the world coordinate system, so the relative 

tion with the routine of FIG. 7. During or after such rotation angles in the second row of the matrix are all 

generation of in-betweens, the sequence of animated 45 °°- Final ly> it is assumed that, for this example, the scale 

frames can be displayed, and modifications to the se- factors f° r the frame are all unity. The limb L* associ- 

quence can be implemented, as represented by block atcd witt ^ exemplary joint is represented by the 

216 and as described in conjunction with the routine of following vector list: 

FIG. 8. The final sequence of frames can then be stored 50 K1 1 0) ,_ { {JJ) , . , „ 

(block 218) and rendered (block 219) by providing ap- 1 a- C * ( ^ 
propriate surface characteristics to the wire frame type 

structures that result from the described procedure. The list sets forth the vertices of the limb as defined in 

Appropriate color rendering can also be implemented, the local coordinate system (FIG. 4). A comma sepa- 

as is known in the art. One or more frame buffers (e.g. 55 rates the points which are joined by lines, so some indi- 

60) and color monitor 25 can be used toward this end. In vidual points may be listed more than once, 

this regard, reference can again be made to the book Reference can now be made to FIG. 4B which repre- 

"Principles of Interactive Computer Graphics", as well ***** ^ same joint J/and limb L,-, but with the position, 

as to U.S. Pat Nos. 4,189,743 and 4,189,744, assigned to orientation, and size and shape of the limb being differ- 

the same assignee as the present application, and to the ^° ent bv virtue of modifications in the matrix values of the 

patents listed in the background portion hereof. examplary joint In particular, the matrix for the joint of 

The position, orientation, and scaling factors of the FIG * * BisQS follows: 
set of local coordinates for a particular joint is defined 

with respect to the coordinate axes of the next higher 65 r 10 l5 10 "| 

priority joint in the hierarchy of joints. In particular, a 45* o* o* 

parametric matrix defining the local coordinate space is L 1 1 2 J 
given by: 



35 



[10 10 10" 
0* 0* 0" 
1 1 1 _ 
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In this case, the origin of the local coordinate axes (Le., 
the joint position) is seen to be at the coordinates (10, 15, 
10), which correspond to the values in the first row of 
the matrix. Also, it is seen that the y, z axes are rotated 
by 45° around the x axis as compared to the orientations 
of the y f z axes in the parent "world" coordinate system. 
Accordingly, the second row of values has a 45" rota- 
tional angle indicated in the x column, and 0° rotational 
angles indicated for rotation around the y and z axes. 
Regarding scaling, the limb of FIG. 4A is seen to be 
doubled in length or "stretched" along the z axis (only). 
Thus a scale factor of 2 is indicated in the z column of 
the third row of the matrix. It will be understood that in 
the present embodiment the representations of the limb 
vectors can remain fixed during a sequence of anima- 
tion, with the changes in position, orientation, and size 
and shape of the limb being achieved by modification of 
the parametric matrix of the joint with which the limb is 
associated. Also, when a modification in a parameter of 
a joint is implemented, it is seen that the limbs of all 
lower priority joints are effectively modified along with 
their parent joints. 

It is known in the computer graphics art that geomet- 
ric transformations can be used in generating an image 
on a two-dimensional screen of three-dimensional 
scenes, Reference can be made, for example, to the 
book, "Principles of Interactive Computer Graphics," 
referred to hereinabove. In the present invention, ma- 
trix transformations are employed to transform the limb 
vectors in a given local coordinate system into equiva- 
lent vectors in the joint coordinate system of next high- 
est priority. The procedure is repeated for successively 
higher priority joint coordinate systems until the limb 
vectors are expressed in terms of the world coordinate 
system. Finally, a perspective transformation is used to 
project the three-dimensional scene onto a two-dimen- 
sional surface (e.g. corresponding to the display screen 40 
surface). This perspective transformation depends upon 
the location, rotational angle and viewing angle of an 
imaginary camera whose position and other parameters 
are variable under operator control, such as by manipu- 
lating one of the joy sticks. 

It is known that if a geometric transformation does 
not deform an object it transforms, then it must be de- 
composable into primitive translation and rotation 
transformations. As described, the inclusion of a scaling 
transformation allows stretching and contraction of 
limbs during motion. The matrix transformation which 
translates a point (x,y,z) to a new point (x',y',z') is 
known to be: 
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where m x , inland m 2 are the components of the transla- 
tion or move in the x, y and z directions respectively, 
and the 1 in each fourth column is the homogeneous 
coordinate. 

Rotation about the z axis (for example), through an 
angle 0 can be achieved with the following transforma- 
tion: 
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cos e 

sin 6 

0 

0 



-sin 6 
cos & 
0 
0 



It can be noted that this transformation matrix affects 
only the values of the x and y coordinates. Similarly 
derived transformation matrices can be set forth for 
rotation about the x and y coordinate axes. 

A scaling transformation, for use to scale dimensions 
in each coordinate direction, is as follows: 



[yyyij-txj^i] 



0 0 

0 0 

*z 0 

0 1 



where s*, and feare, respectively, the scaling factors 
in the x, y and z directions. 

If the translation, composite rotation, and scaling 
matrices are represented by [M], [R], and [S], respec- 
tively, a composite transformation matrix [T], can be set 
forth as: 

[T]=[M]MS] 

This composite transformation matrix corresponds to 
the composite parameter matrix for the joint local coor- 
dinate system first set forth above at (1). 

To transform the vectors of a limb associated with a 
joint J„ which has respective parent, grandparent, etc. 
joints J„- 1, Jn-2, etc., into a main or world coordinates 
system (represented as Jo), the overall transformation 
matrix, PWo] can De represented as the concatenation 
of the various local matrix transformations, as follows: 

Un-«Qi==|Tt]rT2) rr "- l]tT "J 

where the subscripts of the transforms and the joints 
correspond. 

Once expressed, in terms of the world coordinate 
system, a known viewing transformation [V] can be 
used to convert points in camera two-dimensional per- 
spective coordinate system (xc, y£ in accordance with: 

[x&tZ&A = [x*y**wl][y] 

where [V] is known to be a composite viewing transfor- 
mation that is built up from several translations and 
rotations that are determined from the viewing parame- 
ters, and w c is the resulting homogeneous coordinate. 
Reference can again be made, for example, to the previ- 
ously cited book and patents. The transformation to the 
appropriate two-dimensional coordinates (x,y) is 
thereby seen to be achieved by multiplying the points in 
the world coordinate system by the viewing transfor- 
mation and subsequent division by the homogeneous 
coordinate. 

Referring to FIG. 5, there is shown a flow diagram of 
a routine for controlling the display of the three-dimen- 
sional images stored in memory in joint and limb for- 
mat In general, the routine involves the obtainment of 
the two-dimensional (x,y) coordinates of the figures to 
be displayed (for each frame) by concatenation of ma- 
trix transforms, as previously described, to obtain the 
two-dimensional projections of the limb vectors in the 
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main world coordinate system as viewed with an opera- the cursor. The parameter to be modified, i.e. motion, 
tor-selected imaginary camera. The two-dimensional rotation, or scaling, is identified, as represented by the 
projections are displayed on the line drawing display 72 block 614. In an embodiment of the present invention, a 
of system 70. The set-up and multiplication of the matri- keyboard control key is utilized to identify the selected 
ces can be implemented in the general purpose proces- 5 parameter, and a joystick is thereby placed in a mode to 
sor 30 but, more preferably, is achieved by providing control the selected parameter. However, it will be 
the matrix information to the system 70 which is well understood that, if desired, separate joysticks can be 
suited for this purpose. provided for each parameter. In either event, motion of 
In FIG. 5, the block 511 represents the reading of the the joystick is effective to modify the x, y and z compo- 
stored parametric matrix of the next highest priority 10 nents of the selected row of the above-described para- 
joint (beginning with the highest priority joint). In the metric matrix (which, in turn, causes the displayed limb 
present embodiment, the joints are processed in order, appearance to change accordingly, consistent with the 
from highest priority to lowest priority, as this facili- described display routine). The modification of the 
tates the computations. The block 512 is then entered, values in the selected row of the joint's parametric ma- 
this block representing the multiplication of the matrix 15 trix is represented by the block 615. The technique is 
for the current joint by the concatenation of matrix repeated for all joints to be manipulated (block 616), 
transformations for the previous joints of higher prior- and the modified figure is stored in conjunction with the 
ity than the joint currently being processed and then by key frame indication (block 617). The procedure can 
the viewing transformation, consistent with relation- then be repeated, if desired, to add figures to the scene 
ships just set forth above. The latest concatenation of 20 of the key frame and manipulate the figures as desired 
matrix transformation is stored, as represented by the (block 618). In this manner, the individual key frames 
block 513. It will be understood that when the next can be formulated and stored. 

lower priority joint is processed, the stored concatena- FIG. 7 illustrates the routine of the present embodi- 

tion of matrix transformations can be used again so as to mem for generating in-between frames. It is assumed 

reduce the necessary computation. 25 that for a sequence of frames to be in-betweened, there 

A determination is next made (decision diamond 514), are a number of key frame poses for each of one or more 

as to which display mode is active. In the present em- figures, and the object of the in-between generation is to 

bodiment, the operator can elect to display either limbs produce natural or other desired motion of figures in 

which comprise the actual figures, or skeletons which the sequence. The block 711 represents selection of the 

are useful in visualizing the positions of the joints and 30 sequence of frames to be in-betweened. The joint to be 

the statuses of the coordinate axes defined by the joints. treated is selected, (block 712), and the next parameter 

If the figure limbs themselves are to be displayed, the to be interpolated is selected, as represented by the 

block 515 is entered, this block representing the reading block 713. Each of the motion, rotation, and scaling 

of the limb vector coordinates associated with the cur- parameters of the transformation matrices of the current 

rent joint, and the multiplication of each point by the 35 joint are interpolated in the present embodiment, and 

concatenated matrix transformation. If the figure skele- this is done for each of the x, y and z components. The 

tons are to be displayed, a basic orthogonal coordinate x, y or z component is selected (block 721). Next, an 

system representation (Le. f three mutually orthogonal interpolation curve is fit between the selected parameter 

lines), can be multiplied by the concatenated matrix values for the selected component. In an operational 

transformation, so as to obtain properly oriented and 40 embodiment hereof, a cubic curve was fit through the 

scaled local coordinate axes representation at each joint points in a standard in-betweening plot of frame number 

position, as shown in the small sketch near block 516. versus the parameter being interpolated (see eg. FIG 

The block 521 is then entered (from block 515 or 516), 10). Block 723 is then entered, this block representing 

this block representing the storage and display of results the storage of values from the interpolation curve in the 

on line drawing display 72 of system 70. A detennina- 45 transformation matrices of the frames being in- 

tion is then made (diamond 522) as to whether or not the betweened. In particular, the selected one of the nine 

last joint in the frame has been processed. If not, the values in the parametric matrix described above for 

block 511 is reentered. When all joints have been pro- each joint is stored for each of the in-between frames, 

cessed, display of the frame can be repeated, or the next based on the value of such parameter component taken 

frame displayed, depending upon the function being 50 from the cubic interpolation curve. Determination is 

performed. then made as to whether or not all three components 

Referring to FIG. 6, there is shown a flow diagram of have been considered (diamond 724). If not, the loop 

routine for forming and modifying the key frames, as 731 continues. When complete, a determination is made 

represented generally by the block 213 of FIG. 2. The (diamond 725), as to whether or not all parameters have 

key frame to be formed is identified (block 611), and the 55 been processed. If not, the loop 732 continues. When all 

first figure to be placed in the scene represented by the parameters have been processed, determination is made 

key frame being formed is accessed by calling up its (diamond 726), as to whether or not all joints for all 

skeleton file (and the limb file which was previously figures in the sequence of frames have been processed, 

stored in conjunction with the skeleton file), as repre- If not, the loop 733 is continued until all joints have 

sented by the block 612. The joint (and accompanying 60 been processed, as described. In this manner, the in-be- 

limb) to be manipulated is then identified, as represented tween frames are formed and stored. It can be noted, as 

by the block 613. As described hereinabove, the display represented by the block 729, that the limbs associated 

of the figure can be implemented, at the operator's op- with each joint are obtained in conjunction with the 

tion, either in terms of a skeleton of joints or as a full joint in each frame of the sequence, the motion of the 

figure of limbs. A cursor can be conventionally em- 65 limb being completely defined as described above, by 

ployed to identify the joint whose accompanying limb is the matrix transformation values of the associated joint 

to be manipulated. Alternatively, the limb associated In accordance with a feature of the invention, when a 

with the joint to be manipulated can be identified with generated sequence of animation is displayed, the opera- 
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tor can modify the motion of a limb during a continuous 
display of the sequence. This is achieved by accessing 
the parametric transformation matrix for the joint asso- 
ciated with the selected limb, and substituting, during 
display of the sequence, parameter components which 
are selected by the operator (preferably, by manipula- 
tion of a joystick), and are used in place of the previ- 
ously stored parameter components of the joint during 
the frame sequence in question. In the present embodi- 
ment, at each frame of a sequence being displayed, the 
x, y and z components of a selected parameter (move, 
rotation or scale factor) of a selected joint are incre- 
mented by an amount which is proportional to the oper- 
ator selection of x, y and z values using the joystick. The 
routine for effecting this feature is illustrated in FIG. 8. 15 

The frame sequence to be reviewed and modified is 
identified (block 811). The joint and parameter to be 
modified are selected by the operator, as represented by 
block 812. As frames are displayed, the operator manip- 
ulates the joystick to indicate increments in the x, y and 20 
z components of the selected parameter of the selected 
joint The sensing of these increments is represented by 
the block 821. The matrix parameter components are 
modified m accordance with the sensed increments, as 
represented by the block 822. A frame is then displayed 25 
(block 823), in accordance with the display routine of 
FIG. 5. Determination is then made (diamond 824) as to 
whether or not the full sequence of frames has been 
displayed. If not, the next frame of the sequence is pro- 
cessed (block 825) by reentering block 821, and the loop 30 
is continued as each frame, with the modified limb, is 
displayed. The procedure can then be performed for 
any selected joint or parameter, under operator control. 
Also, it will be understood that both the old and new 
sequences of matrices for the selected joint can be saved 35 
so that the operator can decide what is to be used in the 
finally compiled frame sequence. 

The motion of a limb during sequence of frames can 
- also be varied by specific modification of the interpola- 
' tion curve for each component of each parameter of 40 
transformation matrix of the joint associated with the 
particular limb. For example, in the present embodi- 
ment, the interpolation curve can be displayed, such as 
in FIG. 10 which shows a curve for the x component of 
position ("move") fitted to four key frames. The com- 45 
puted in-between frame x positions are shown as hollow 
dots and the key frame positions are shown as solid dots. 
The solid curve is the original interpolation curve, and 
the dashed line curve represents an operator-modified 
curve which can be input, for example, via the data 50 
tablet Preferably, the old and new curves are both 
saved so that the motion of a particular limb can be 
modified until the operator is satisfied. It will be under- 
stood that similar operation can be performed for the x, 
y and z components of rotational orientation and scale 55 
factor, as well as for the other components of position. 

FIG. 9 illustrates the routine for accessing and modi- 
fying the interpolation curve as described. The operator 
first indicates the sequence of frames to be reviewed 
(block 911). The joint and parameter component to be 60 
modified are then identified (block 912) and the corre- 
sponding interpolation curve (see FIG. 7 routine) is 
displayed, as represented by the block 913. The opera- 
tor can then draw the modified interpolation curve (or 
input the information in any other desired way), as 65 
represented by the block 914. The modified curve val- 
ues can then be stored, and the sequence of frames re- 
viewed with the new curve. 
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The invention has been described with reference to a 
particular embodiment, but variations within the spirit 
and scope of the invention will occur to those skilled in 
the art. For example, it will be understood that key 
5 frames can be defined in terms of individual figures 
which have key poses in those frames, so that different 
frames will be considered key frames with regard to 
different figures. Also, the invention is not dependent 
upon any particular type of coordinate transformations 
10 or viewing transformations being used, and additional 
techniques, such as for handling clipping can be imple- 
mented, as is known in the art While the invention is 
described in terms of a monochrome system, it has ap- 
plication to color systems as well. It will also be under- 
stood that special purpose hardware implementations of 
various described portions of the system could be used 
if desired. Finally, it can be noted that means, such as 
television camera 65, can be used to obtain images of 
figures and basic positions and geometries of three-di- 
mensional figures can be developed using these images 
as an aid. 
I claim: 

1. A method for generating a sequence of video 
frames representative of three-dimensional animation, 
comprising the steps of: 

storing a plurality of key frames, each including a 
common figure having one or more joints, each 
joint having associated therewith a set of vectors 
defining a limb, each joint being defined in each 
key frame by operator-controllable parameters 
which determine in three dimensions the position, 
rotational orientation, and scale factors of a local 
coordinate system in which the limb vectors are 
placed; 

generating a plurality of in-between frames which 
include said common figure having one or more 
joints and associated limbs corresponding to the 
joints and associated limbs of said common figure 
in successive key frames, the parameters of the 
joints of an in-between frame being obtained as 
follows: 

(a) performing an interpolation, in each dimension, of 
the position, rotational orientation, and scale fac- 
tor, for the local coordinate system associated with 
a joint, such that the interpolated parameter values 
of the joint lie between the parameter values associ- 
ated with the corresponding joints of the succes- 
sive key frames; and 

(b) repeating step (a) for other joints of said common 
figure. 

2. The method as defined by claim 1, wherein the 
joints of each figure are arranged in a hierarchical or- 
der, and wherein the positional coordinates and rota- 
tional orientations of the local coordinate system for a 
particular joint are determined with respect to the local 
coordinate system of the next higher joint in the hierar- 
chy. 

3. The method as defined by claim 1, further compris- 
ing the step of operator-controlling the interpolation, 
during generation of the in-between frames, so as to 
change the motion of a figure limb. 

4. The method as defined by claim 2, further compris- 
ing the step of operator-controlling the interpolation, 
during generation of the in-between frames, so as to 
change the motion of a figure limb. 

5. The method as defined by claim 1, further compris- 
ing the steps of determining the two-dimensional coor- 
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dinates of said figure as viewed from an imaginary cam- of vectors defining a limb, each joint being defined 

era perspective, and displaying the figure on a screen. in each key frame by operator-controllable param- 

6. The method as defined by claim 2, further compris- eters which determine in three dimensions the posi- 
ing the steps of determining the two-dimensional coor- tion, rotational orientation, and scale factors of a 
dinates of said figure as viewed from an imaginary cam- 5 local coordinate system in which the limb vectors 
era perspective, and displaying the figure on a screen. are placed; and 

7. The method as defined by claim 3, further compris- means for generating a plurality of in-between frames 
ing the steps of detenriining the two-dimensional coor- which include said common figure having one or 
dinates of said figure as viewed from an imaginary cam- more joints and associated limbs corresponding to 
era perspective, and displaying the figure on a screen. 10 the joints and associated limbs of said common 

8. The method as defined by claim 4, further compris- figure in successive key frames; 

ing the steps of determining the two-dimensional coor- said means for generating in-between frames includ- 
dinates of said figure as viewed from an imaginary cam- ing means for obtaining the parameters of the joints 

era perspective, and displaying the figure on a screen. of an in-between frame by performing an interpola- 

9. The method as defined by claim 7, wherein said 15 tion, in each dimension, of the position, rotational 
step of controlling the interpolation comprises opera- orientation, and scale factor, for the local coordi- 
tor-controlling, during display of the figure in said se- nate system associated with each joint, such that 
quence of video frames, the three-dimensional compo- the interpolated parameter values of the joint lie 
nents of position, rotational orientation, or scaling of the between the parameter values associated with the 
joint associated with said limb. 20 corresponding joints of the successive key frames. 

10. The method as defined by claim 8, wherein said 16. Apparatus as defined by claim 15, wherein the 
step of controlling the interpolation comprises opera- joints of each figure are arranged in a hierarchical or- 
tor-controlling, during display of the figure in said se- der, and wherein the positional coordinates and rota- 
quence of video frames, the three-dimensional compo- tional orientations of the local coordinate system for a 
nents of position, rotational orientation, or scaling of the 25 particular joint are determined with respect to the local 
joint associated with said limb. coordinate system of the next higher joint in the hierar- 

11. The method as defined by claim 9, wherein said chy. 

operator-controlling of three-dimensional components 17. Apparatus as defined by claim 15, further corn- 
comprises manipulating a joystick during said display of prising means for operator controlling the interpolation, 
the figure in said sequence of video frames. 30 during generation of the in-between frames, so as to 

12. The method as defined by claim 10, wherein said change the motion of a figure limb, 
operator-controlling of three-dimensional components 18. Apparatus as defined by claim 16, further corn- 
comprises manipulating a joystick during said display of prising means for operator controlling the interpolation, 
the figure in said sequence of video frames. during generation of the in-between frames, so as to 

13. The method as defined by claim 1, further com- 35 change the motion of a figure limb. 

prising the step of displaying a curve representative of a 19. Apparatus as defined by claim 15, further corn- 
component of the three-dimensional interpolation of the prising means for determining the two-dimensional co- 
position, rotational orientation, or scale factors of a ordinates of said figure as viewed from an imaginary 
figure limb; operator-modifying said curve; and storing camera perspective, and means for displaying the figure 
joint parameters corresponding to values of the modi- 40 on a screen. 

fled curve; so as to modify motion of said figure limb 20. Apparatus as defined by claim 16, further com- 
during said sequence of video frames. prising means for determining the two-dimensional co- 

14. The method as defined by claim 2, further com- ordinates of said figure as viewed from an imaginary 
prising the step of displaying a curve representative of a camera perspective, and means for displaying the figure 
component of the three-dimensional interpolation of the 45 on a screen. 

position, rotational orientation, or scale factors of a 21. Apparatus as defined by claim 17, further corn- 
figure limb; operator-modifying said curve; and storing prising means for determining the two-dimensional co- 
joint parameters corresponding to values of the modi- ordinates of said figure as viewed from an imaginary 
fied curve; so as to modify motion of said figure limb camera perspective, and means for displaying the figure 
during said sequence of video frames. 50 on a screen. 

15. Apparatus for generating a sequence of video 22. Apparatus as defined by claim 18, further corn- 
frames representative of three-dimensional animation, prising means for determining the two-dimensional co- 
comprising: ordinates of said figure as viewed from an imaginary 

means for storing a plurality of key frames, each camera perspective, and means for displaying the figure 
including a common figure having one or more 55 on a screen, 
joints, each joint having associated therewith a set * * * * * 
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REEXAMINATION CERTIFICATE 
ISSUED UNDER 35 U.S.C. 307 

THE PATENT IS HEREBY AMENDED AS 
INDICATED BELOW. 

Matter enclosed in heavy brackets [J appeared in the 
patent, but has been deleted and Is no longer a part of the 
patent; matter printed In italics indicates additions made 
to the patent 

AS A RESULT OF REEXAMINATION, IT HAS 
BEEN DETERMINED THAT: 

Claims 1-22 arc cancelled. 

New claims 23-36 are added and determined to be 
patentable. 

23. The method as defined by claim L wherein said sets 
of vectors define three-dimensional limbs* and further 
comprising the steps of applying said sets of vectors defining 
said limbs to the joints of said generated in-between frames, 
and displaying the generated in-between frames as a con- 
tinuous video sequence to view the motion of the three-di- 
mensional limbs. 

24. The method as defined by claim 23, wherein said 
displaying step includes determining the two-dimensional 
coordinates of said limbs as viewed from an imaginary 
camera perspective and displaying the figure as a continu- 
ous sequence of wire frame line drawings on a two-dime- 
sional display. 

25. The method as defined in claim 24, further compris- 
ing modifying in-between frames of said sequence of 
frames, and subsequently rendering the modified sequence 
of frames. 

26. The method as defined by claim 23, further compris- 
ing modifying in-between frames of said sequence of 
frames, and subsequently rendering the modified sequence 
of frames. 

27. The method as defined in claim 26, wherein said 
modifying includes operator-controlling the interpolation 
during said display of said generated sequence of frames so 
as to change the motion of a figure limb, 
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28. The method as defined in claim 23, further compris- 
ing operator-controlling the interpolation during said dis- 
play of said generated sequence of frames, so as to change 
the motion of a figure limb. 
5 29. The method as defined by claim 28, wherein the 
joints of each figure are arranged in a hierarchial order, 
and wherein the positional coordinates and rotational ori- 
entations of the local coordinate system for a particular 
joint are determined with respect to the local coordinate 
to system of the next higher joint in the hierarchy. 

3a The method as defined in daim 28, wherein said 
operator-controlling the interpolation comprises manipu- 
lating a joystick during said display of said generated se- 
quence of frames. 
15 31 The method as defined in claim 23, wherein the 
joints of each figure are arranged in a hierarchial order, 
and wherein the positional coordinates and rotational ori- 
entations of the local coordinate system for a particular 
joint are determined with respect to the local coordinate 
20 system of the next higher joint in the hierarchy. 

32 Apparatus as defined in claim 15, wherein said sets 
of vectors define three-dimensional limbs, and further 
comprising means for applying said sets of vectors defining 
said limbs to the joints of said generated in-between frames, 
25 and means for displaying the generated in-between frame 
as a continuous video sequence to view the motion of the 
three-dimensional limbs, 

33. Apparatus as defined in claim 32, further comprising 
means for modifying in-between frames of said sequence of 

JO frames, and means for subsequently rendering the modi- 
fied sequence of frames. 

34. Apparatus as defined by claim 32, wherein the joints 
of each figure are arranged in a hierarchical order, and 
wherein the positional coordinates and rotational orienta- 
ls tions of the local coordinate system for a particular joint 

are determined with respect to the heal coordinate system 
of the next higher joint in the hierarchy. 

35. Apparatus as defined by claim 34, further compris- 
ing means for operator-controlling the interpolation during 

40 said display of said generated sequence of frames, so as to 
change the motion of a figure limb. 

36. Apparatus as defined in claim 32, further comprising 

means for operator-controlling the interpolation during 

said display of said generated sequence of frames, so as to 

45 change the motion of a figure limb. 
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